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LONGER-TERM STRATEGY OE THE DEVELOPMENT OE PISA 



Introduction 

1 . The original data strategy on which PISA was based suggested that, after the completion of the 
first three assessments in 2000, 2003 and 2006, the cycle would repeat itself with three-yearly assessments 
in the areas of reading, mathematics and science [see doc. ref. DEELSA/ED/CERI/CD(97)4, which 
provides the original data strategy on which PISA was based] . However, in light of newly emerging policy 
priorities and the experience gained with PISA so far, the PISA Governing Board began in March 2004 to 
review the objectives and design of the PISA data strategy for the period 2009 and beyond. 

2. At its last meeting on 7-9 March 2005, the PISA Governing Board discussed a first strategy paper 

addressing the longer-term development of PISA, which been prepared on the basis of an informal 
consultation of Member countries on future perspectives for PISA [doc. ref. 

EDU/PISA/GB/SDG(2005)1] . This strategy paper had set out options to innovate - rather than 
substantially alter - the design of PISA, seeking to maximise synergies with other OECD and non-OECD 
data sources and ensuring that PISA would remain manageable, draw on up-to-date methodologies, and 
stay within its existing financial framework. 

3. Polio wing this meeting, a formal consultation of Member countries was then undertaken to 
determine national priorities for the further development of PISA [doc. refs. EDU/PISA/GB/SDG(2005)2 
and 3]. Country responses to this consultation fell essentially into three groups: 

• A group of countries who preferred to maintain PISA as a “lean” instrument, based on the 
existing design and administration on a three -yearly basis; 

• A group of countries wishing to expand PISA but administer the survey on four-yearly cycle; and 

• A small group of countries wishing to reduce the scope or maintain the scope of PISA but 
implement the survey on a four-yearly cycle. 

4. The PISA Strategic Development Group', then met on 27-29 June 2005 to review the individual 
country responses^ as well as a summary of country positions on the various dimensions of the strategy 
[doc. ref. EDU/PISA/GB/SDG(2005)4REV] and prepared proposals for their consolidation as well as a 
range of other recommendations for the development of the longer-term development of PISA [doc. ref. 

EDU/PISA/GB/SDG/M(2005)2] . 



Current members: Lorna BERTRAND (United Kingdom), Gerard BONNET (Erance), Satya BRINK 
(Canada), Giacomo ELIAS (Italy), Eelipe MARTINEZ (Mexico), Jules PESCHAR (The Netherlands), 
Alette SCHREINER Norway), Elois SCOTT (United States), Peter Vari (Hungary), Ryo WATANABE 
(Japan). Chair: Jochen SCHWEITZER. Special advisor: Eugene OWEN 

Individual country responses can be found on the Electronic Discussion Group of the PISA Governing 
Board. 
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5. These proposals and recommendations were then sent to countries with a request for review and 
approval. During this written consultation, countries generally welcomed the proposals and 
recommendations from the Strategic Development Group hut also raised a number of issues, which 
included requests: 

• For greater clarity on the implications of the modular design for analysis and reporting (with the 
concern that international comparahility will become more complex with different sets of 
countries included in different sets of analyses and indicators). 

• To separate the grade -based component from the development of additional context 
questionnaires and the link to the teacher survey (currently, these are all integrated in one 
module). 

• To open the discussion on the most appropriate age group for the assessment of a younger age 
cohort, with 1 1 years offered as another possibility to the age of 9 years, that had so far been 
discussed. 

• To carefully assess the implications of any changes to the PISA test design and objections to 
reducing individual assessment time through expanded sample sizes. 

• For more conceptual and methodological work on new assessment domains before closing off the 
possibility to develop these. 

• To clarify why the proposal to expand the context questionnaires should imply giving more 
weight to learning strategies and questions on student destinations etc. 

• To consider whether all modules should already be developed for PISA 2009 or whether to 
sequence the development of these modules in some way. 

• To clarify the need to maintain the Open Forum. 

• To clarify the value of linking the TAG closer to the PISA Governing Board. 

• To review the balance between data development and analysis and the roles and responsibilities 
of the different actors involved in this work, including the possibility to facilitate publications 
under different authorship. 

6. At its meeting on 19-20 September 2005, the PISA Strategic Development Group took these 
issues up and introduced the following main changes to the proposed data strategy: 

• A “time dimension” has been incorporated into the development of the optional modules, such 
that these would now be progressively introduced over multiple assessments, rather than at once 
in the 2009 assessment. 

• The revised paper gives greater emphasis to pursuing innovation within the existing assessment 
areas as well as the development of new assessment areas. 

• The revised paper now deals with the implications of the modular design for analysis and 
reporting. 

• The revised design of Module 3 separates the grade -based component from the link to the teacher 
survey. 

• The strategy paper opens up possibilities for changing the target age for the younger age group. 
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• The management structure has been revised further with the aim to establish closer linkages 
between the PISA Governing Board and the Technical Advisory Group. In particular, a report 
from the Technical Advisory Group would now be presented to the PISA Governing Board prior 
to each meeting that would clarify the implications of technical issues on policies, analytic 
outcomes and the financing of PISA. 

7. The revised strategy, as presented in this document, is organised as follows: 

• The first part of the paper describes the revised future PISA data strategy, including an updated 
summary of country responses to the written consultation. 

• The second part examines how the core and optional modules would be linked to the key analytic 
objectives of PISA, namely to facilitate international comparisons on: 1) the quality of learning 
outcomes; 2) equality in learning outcomes and equity in learning opportunities; 3) the 
effectiveness and efficiency of educational processes; and 4) the impact of learning outcomes on 
social and economic well-being. 

• The last part of the document presents possible modifications to the PISA management structure. 

8. The PISA Governing Board will need to: 

• REVIEW and EINALSE the longer-term strategy for PISA. 

Design of the future PISA data strategy 

9. During discussions and a written consultation, countries explored modifications to the overall 
design of the strategy, including the choice of subject areas and target populations to be assessed, the 
balance between assessment and analysis, and the implementation and frequency of successive assessments. 

10. Areas where a majority of countries had expressed interest in changes to the PISA design 
included (presented in descending order of country ratings): the addition of a younger age cohort to PISA 
in order to compare growth in student performance (considered relevant or very relevant by 84% of the 28 
responding OECD countries); giving more emphasis to student-level contextual data (considered relevant 
or very relevant by 82% of respondents) or even reducing assessment time to extend the student context 
questionnaires (considered relevant or very relevant by 57% of the respondents); the integration of a grade- 
based assessment component (considered relevant or very relevant by 64% of the respondents); the 
development of ICT literacy as a new assessment domain (considered relevant or very relevant by 61% of 
respondents); and developing contextual data on teachers and teaching by linking PISA and the OECD 
teacher survey (considered relevant or very relevant by 59% of the respondents). 

11. With regard to the frequency of successive assessments, countries were divided between 
continuing to administer PISA on a three-yearly basis and changing the gap between successive 
assessments to four years. 46% were in favour of the continuation of the current three-yearly cycle, 
although 18% of these noted that they would also find a four-yearly administration cycle acceptable. One 
country (4%) voted for each two three-yearly assessments to be followed by one four-yearly assessment^. 
50% of the countries were in favour of changing the frequency to four years, although 11% of these 
countries noted that they would also find a three -yearly administration cycle acceptable. 



Also one of the communities of Belgium voted for this option. However, since the other community voted 
for the three yearly cycle with acceptance of 4 years, both communities have been classified under this 
option for the purpose of calculating percentages. 
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12. Nevertheless, while there was considerahle interest among some countries to change the basic 
design and/or frequency of PISA assessments, a sizeable group of countries strongly opposed any such 
design changes as well as changes to the frequency of PISA assessments on the basis of which they had 
joined PISA in 1997. Since these are aspects where the PISA ground rules require consensus among 
delegates to make any changes [doc ref. C(1997)176], the Strategic Development Group has now 
established a strategy that: 

• Maintains the basic design and frequency of the current PISA cycles and builds on the success of 
PISA through improvements within the survey's existing design and structure. 

• Pursues changes and extensions in the form of additional modules that would be designed, 
implemented and financed by interested countries, with the options technically compatible so that 
countries could choose to participate in as many modules that are relevant to their national 
context. 

13. The PISA Governing Board could consider the outcomes of each module at the completion of 
each assessment cycle and, at that point, decide whether to maintain these optional or to integrate them into 
the core PISA instruments for subsequent cycles. In other words, it is proposed that any new aspects to 
PISA be pursued as an option first before being proposed for integration into the core of subsequent PISA 
assessments"^. 

14. To facilitate such an ongoing evaluation and development process, and to strengthen 
relationships between PISA and other OECD and non-OECD data sources, countries could consider 
establishing the Strategic Development Group as a permanent advisory group to the PISA Governing 
Board. 

15. The set of modules described below was established to reflect the choices that countries made 
during the written consultation. It was assumed that areas which less than half of the countries considered 
relevant would only be pursued in the form of national options rather than being part of the international 
study design which interested countries would collectively develop, finance and implement through 
internationally agreed procedures. 

1 6. The PISA analysis and reporting plans would be primarily designed around the core module, with 
results from the optional modules reported separately either in specific sections of the main reports or in 
separate reports. 

17. Several of the proposed optional modules would require considerable further development and 
the Strategic Development Group therefore suggests to introduce these progressively over multiple 
assessments. The proposed timing is summarised in the table below and described in more in subsequent 
sections of the document. 



In some areas, the outcomes and their utility for policy purposes may not become apparent for immediately 
so that there may be lag of more than once assessment cycle before these would be implemented 
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Table 1: Timing and sequencing for the development of the optional modules 





Optional Module 1: 

Computer delivered assessment 
and assessment of ICT literacy 


Optional Module 2: 

Comparing progress in education 
systems 


Optional Module 3: 

Linking student performance 
with the instructional context 


2009 


Review experience gained with 
CBAS in PISA 2006. 

Examine psychometric 
compatibility of performance data 
obtained through paper-and-pencil 
assessments and computer- 
delivered technologies. 

Develop framework for assessing 
ICT literacy. 


Explore three levels of linkages 
between PISA and lEA studies, 
ranging from the co-ordination of the 
assessment frameworks, through the 
co-ordination of the target 
populations, up to the co-ordination 
of the samples and cohorts. 

If link with lEA studies does not 
satisfy analytic objectives of OECD 
countries or is not cost-effective, 
establish data strategy to survey 
younger cohort, which involves, 
identifying the most appropriate 
target age for the younger age group, 
establishing the subject areas for 
which progress is to be measured, and 
defining the methods for measuring 
progress. 


Develop comparisons of the 
instructional context of learning 
outcomes through the establishment 
of a supplementary grade/class- 
based assessment (Module 3A). 

Extend the student and school 
context questionnaires on 
instructional context, and collect 
system-level data (Module 3B). 

Extend the sampling procedures to 
facilitate link with OECD survey on 
teachers, teaching and learning, as 
well as the exploration of other data 
collection methods and in-depth 
studies seeking to relate student 
performance and teacher data 
(Module 3C). 


2012 


Develop computer-delivered 
assessments in the regular PISA 
assessment areas (Module lA). 

Establish an assessment of ICT 
literacy (Module IB). 

Develop adaptive assessment 
technologies (Module 1C). 


Implement assessment of younger age 
cohort. 


Continue with linking student 
performance data with the 
instructional context. 


2015 


Continue to implement ICT 
delivery and assessment of ICT 
literacy. 


Continue to implement assessment of 
younger age cohort. 


Continue with linking student 
performance data with the 
instructional context. 



18. Beyond these modules, the Strategic Development Group has recommended that the OECD 
should facilitate bilateral or multilateral co-operation among countries in further areas of development, 
including the articulation of international and national test components with the aim to allow countries to 
relate performance in the internationally assessed competency areas to performance in areas that are 
considered important nationally, including the establishment of multi-lateral assessment components 
among countries with common cultural or educational interests. 

Core module 

19. The basis for future PISA assessments would be a core module to be administered by all 
countries that would: 

• Remain focussed on assessing the cumulative yield of education at the age of 15 years, as the 
highest age at which participation in formal education in OECD countries is still largely 
universal, and which provides a reference point that is definable and external to education 
systems such that student performance can be compared across countries in valid and reliable 
ways. 



7 





EDU/PISA/GB(2005)21 



• Continue to assess to what extent education systems succeed in ensuring that young adults 
acquire some of the key competencies and dispositions believed to contribute to the foundations 
for further learning and a successful transition into adult life (to this end, PISA would remain 
focussed on the capacity of students to use what they have learned in the subject areas of 
language, mathematics and science, rather than being limited to assessing the reproduction of 
subject matter knowledge in these areas). 

• Continue to be administered on a three-yearly basis, starting again with reading as the major 
assessment area in 2009. 

• Build on and improve on the existing frameworks and instruments in order to reflect scientific 
innovation, to improve the relevance of the assessments to participating countries, to optimise the 
measurement of trends in all three areas, and to better match the difficulty of the PISA tests to the 
national ability distribution in countries with particularly high or low performance. 

• Include one developmental assessment domain in each assessment that would be chosen and 
developed by participating countries. This could be an assessment of cognitive performance, like 
the assessment of problem-solving competencies had been in the PISA 2003 assessment, or the 
direct or indirect assessment of student dispositions to learning, such as the self-assessment of 
learning strategies in 2000, or the assessment of student attitudes towards science in 2006. The 
development of this additional assessment domain would be pursued through a corresponding 
developmental track. 

• Include questionnaires to contextualise and interpret observed performance differences and to 
provide policy-makers with a better understanding of cross-country differences in the 
performance of students and schools, including a student questionnaire, a school questionnaire, 
and an optional parent questionnaire. 

20. Drawing on the findings of the INES Network A’s technical review panel, bidders would be 
encouraged to seek technical improvements for PISA in areas such as the balance between major and 
minor assessment areas; the amount of individual assessment time; the PISA test design; the balance 
between assessment and contextual components; and the measurement of performance and the quality of 
trend data. However, the PISA Governing Board would assess any proposed technical improvements 
against their implications in terms of the extent to which they ensured the coherence of the PISA database 
and allowed to maintain consistent trend lines. 

Optional Module One: Computer-delivered assessment and assessment of ICT literacy 

21. As part of PISA 2006, 13 countries are currently piloting instruments for the computer-based 
delivery of PISA assessments. Such instruments could pave the way not only for better coverage of aspects 
of the PISA frameworks that are very difficult to capture with static paper-and-pencil documents but also 
for improved efficiency of the assessment process, both in reducing operational costs and student response 
time and, in the longer term, by targeting the assessments more effectively across and within countries. The 
annex of this document outlines the expected gains as well as resource implications. 

22. Only 19% of the respondents considered that the development of a stand-alone computer- 
delivered platform should become an integral part of the core PISA component (Question 8.1). However, 
adding to this the countries that stated that a computer-delivered platform should be considered on the 
basis of the technical infrastructure that exists in schools or on a web-based platform (Question 8.2), 
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bringing the proportion of countries interested in further work in this area to 81%^. Only 19% considered 
that work in this area would not be a priority (Question 8.3). 

23. The optional Module One would, over time, establish a computer-delivered assessment as well as 
an assessment of ICT literacy through the following developmental phases: 

• The first phase, proposed for the 2009 assessment, would include a review of experience gained 
through the PISA 2006 computer-delivered science assessment and relevant national computer- 
delivered assessments as well as an examination of the psychometric compatibility of 
performance data obtained through paper-and-pencil assessments and computer-delivered 
technologies; 

• The second phase, also proposed for the 2009 assessment, would include an examination of the 
extent to which comparable performance estimates can be obtained from using existing national 
computer infrastructures for the delivery of future web-based PISA assessments, which 63% of 
the responding countries have made a condition for considering computer-based assessments in 
future PISA assessments. 

• The third phase would consist in the development of a framework for assessing ICT literacy. 

• If these initial phases are successful, the development of computer-delivered assessments in the 
regular PISA domains {Module One A) as well as the establishment of an assessment of ICT 
literacy {Module One B) could then both be explored. In other words, a decision on whether to 
pursue the implementation of a computer-based assessment of the PISA test domains, the 
development of a new ICT literacy assessment, or both would be postponed until the 2012 
assessment. 

• Should the ICT-delivered components become an integral part of the PISA data strategy, then 
their further extension to adaptive assessment components could be explored (Module One C). 
These would allow student performance at the top and bottom ends of the PISA assessment 
scales to be better captured, particularly in countries with high or low overall performance, and 
would also result in considerable reduction in individual student assessment time. This is 
proposed for the 2012 assessment. 

24. The development of a computer-based assessment infrastructure {Module One A) has the 
potential to add value to PISA at a number of levels, including the potential to broaden the assessment 
domains and to elaborate PISA at the strategic level in the long term. 

25. The pilot undertaken as part of the PISA 2006 science assessment has shown, for example, that 
certain kinds of scientific thinking can be assessed in computer-based form that cannot be assessed through 
time -limited pencil and paper tests. For example, scientific experiments can be simulated, and the approach 
taken by students to scientific reasoning and experimentation can be assessed. Students’ capacity to design 
experiments, to control the interactions among variables, and to make and test predictions can be directly 
assessed. The response options presented to students in multiple-choice items can take a more varied and 
stimulating form, requiring students to think more critically and scientifically about the questions posed. In 
short, the capacity for interaction between the student and the test question is increased, thereby expanding 
the kind of scientific thinking that can be assessed. The pilot has also shown that the use of dynamic 



One of the Belgian communities had stated that it would consider a computer delivered assessment if this 
could be based on existent school infrastructures. However, the other Belgian community had assigned this 
no priority. The latter response has been used for the calculation of percentages. 
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stimulus enables a number of contexts to be used and a number of scientific areas to be covered that would 
be impossible, or very difficult, with a paper-based test, e.g. change over time. Video can be used to 
represent real-time change, which is very difficult to achieve in a paper and pencil test. Similarly, motion 
can be very easily conveyed using video and animations. This enables various scientific areas, for example 
physics, chemical reactions, earth science and astronomical phenomena, to be demonstrated and assessed. 

26. More generally, a computer-based assessment component can substantially reduce the reading 
load imposed by the test items. This means the test items target the underlying scientific knowledge better, 
and the reading hurdle imposed on students before they are able to demonstrate their science knowledge is 
much less of an impediment. 

27. Experiences with the pilot also suggest that student motivation is increased. Eor many students, 
use of computers has become a normal part of their day-to-day life. Using a computer in a science test is 
motivating to the extent that it fits with students preferred mode of carrying out tasks. 

28. Eurthermore, behavioural data can be captured that cannot be collected using paper-based tests. 
Eor example, the time students actually take on an item, the sequence of actions they take as they interact 
with an item, the number of times they revisit an item and the way they navigate through a test, are all 
examples of information about test-taking behaviour that can easily be captured through the test 
administration software. This information can potentially be used in subsequent test construction, and may 
be useful information for interpreting student scores. 

29. Einally, there are practical advantages of the computer-based assessment component that can help 
to improve the efficiency and reduce costs of the test administration process: 

• The computer-delivered approach can be used to tailor the tests to align with the actual 
performance levels of students, thus avoiding situation where students are confronted with test 
items that are too easy (and thus potentially boring) for them, or with test items that are too 
difficult for them (and thus potentially frustrating) {Module One C). By matching the difficulty 
of the tasks given to students with their actual performance level, measurement can be improved 
and testing time reduced. This is of particular advantage in countries with overall high or low 
performance. 

• Data entry costs for this part of the assessment are dramatically reduced. All data entry is 
automated; with data capture occurring through the software, and being transmitted electronically 
to the Consortium database. In addition, response coding costs for this part of the assessment are 
eliminated. 

• Translation procedures are improved. The translation software which has been developed is web- 
based, which is a more efficient means of communication compared to the present situation. The 
capacity built in to the translation software for double translation, reconciliation of translations, 
and verification of translations will ensure better consistency. 

• Quality assurance of the test instruments themselves are enhanced. The test instruments are 
compiled electronically, using standard programmed processes which apply identically to all 
participants, with a low risk of error. 

30. With regard to the assessment of ICT literacy {Module One B), the framework for the assessment 
of ICT literacy that has been developed through the INES Network A as part of the call for tender for the 
PISA 2006 assessment [see Annex B of document OECD/EXD/PCM/EDU(2003)28] describes the 
analytical benefits an assessment of ICT literacy would entail. 
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Optional Module Two: Comparing progress in education systems 

31. Beyond an examination of trends in learning outcomes over time (pursued through the core 
module) the policy relevance of PISA will he enhanced hy adding to PISA’s capacity to benchmark student 
performance across countries the possibility to compare learning progress within education systems, 
through assessing student knowledge and skills at different stages in the education system. This will be 
achieved through co-ordinated assessments of learning outcomes and associated contextual factors at 
different levels of education. 

32. Such analyses would not need to be limited to examining growth in student performance: 
assessing multiple age levels could also provide insight into how students’ motivation and learner 
characteristics evolve in different countries as students progress through education systems. Potentially, 
they could also provide insight into which instructional and systemic factors are associated with such gains 
in performance and changes in student attitudes towards learning. 

33. Only 30% of the respondents considered the addition of an older cohort to PISA relevant or very 
relevant. (Question 3.1). In contrast, 84% of the respondents considered an assessment of a younger age 
cohort relevant or very relevant (Question 3.2). 

34. 27% of the countries that were interested in a younger age group stated that the assessment of a 
younger age group should be made an integral part of the PISA assessment design (Question 3.2.1), 55% 
suggested that this should be accomplished by linking PISA with lEA assessments if these assessments can 
be made technically compatible (Question 3.2.2), while 27% suggested that the assessment of a younger 
age group should only be pursued through lEA assessments (Question 3.2.3)®. 

35. The optional Module Two would seek to establish comparisons of student progress in education 
systems by benchmarking growth in student performance, possibly between the ages of 9 or 11 years 
(which were both mentioned as potential reference points for the younger age cohort during the written 
consultation), on the one hand, and 15 years, on the other. It would be conceivable to pursue the 
development of Module Two in the following phases: 

• The development of Module 2 would begin with an examination of the extent to which PISA and 
the lEA studies could be meaningfully linked. This examination would explore three levels of 
linkages ranging from a co-ordination of the assessment frameworks, through the co-ordination 
of the target populations, up to the co-ordination of the samples and cohorts. 

- If co-ordination can only be achieved at the level of the assessment frameworks, then 
measures of performance growth could not be developed, but it may become possible to 
relate the description of student performance in both types of studies. 

- If the target populations could be co-ordinated as well, through the addition of a grade-based 
component in PISA and/or the addition of age-based sampling components in lEA studies, 
then it would be conceivable to establish measures of growth based on the comparison of 
synthetic cohorts. Such a link could aim for the psychometric measurement of progress, by 
embedding link items between the assessments of both age groups, or it could simply 
compare relative performance of countries on two different assessments that are both judged 
appropriate by countries for the respective age cohorts. 



The reason why these percentages add up to more than 100 is that some countries (or regions thereof) gave 
multiple responses to this question. This will still need to be consolidated in consultation with the countries 
concerned. 
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- If it would also be possible to co-ordinate the cohorts assessed and to link the timing of both 
survey cycles correspondingly, then this would also allow the development of cohort growth 
measures, a requirement for any assessment of the impact of educational policies over time. 

• If the results of this examination meet the analytic objectives of OECD countries, the two 
organisations would proceed to co-ordinate their surveys correspondingly and the outcomes 
would flow into the respective analyses and reports. If the results of this examination do not 
satisfy the analytic objectives of OECD countries or do not result in cost-effective solutions, they 
would establish a PISA strategy for collecting data for a younger cohort. This would involve 
identifying the most appropriate target age for the younger age group, establishing the subject 
areas for which progress is to be measured, and defining the methods for measuring progress. To 
allow for measures of cohort progress, assessments of 9-year-olds would be implemented 
simultaneously with assessments of 15-year-olds, such that the cohort of 9-year-olds would be 12 
years old at the subsequent PISA assessment and 15 years old at the PISA assessment after that. 
If countries choose 11 years as the target population instead, the assessment of the younger 
cohort would be implemented one year prior to the PISA assessment, such that the cohort of 11- 
year-olds would be 12 years old at the subsequent PISA assessment and 15 years old at the PISA 
assessment after that. 

• This timing would allow the assessment of a younger age cohort to be implemented from the 
2012 PISA assessment onwards. 

36. Adding a younger target population would not only satisfy the specific information needs of 
authorities and educators in charge of primary education; it would also greatly help with understanding the 
long-lasting impact of earlier education on achievement at the end of compulsory schooling. In this respect, 
a younger cohort may contribute even more to the explanation of PISA country level results than the 
inclusion of a grade -based component or a teacher questionnaire for the 15 years-old population. Eor many 
reasons, the PISA study is less suitable for the investigation of classroom-level circumstances and their 
relationship with achievement than for the exploration of school-level and system-level factors. In the 
previous PISA studies, it could be actually seen that some of the most interesting findings were related to 
global factors such as school climate, school intake, school organisation and educational system structure 
(for example, the contrast between comprehensive and tracked systems). One can similarly anticipate that 
collecting information on the global characteristics of the primary schools and primary school systems in 
each country would add important dimensions to the description of factors that contributed to shaping the 
achievement patterns observed at age 15. 

37. Eurthermore, for countries interested in having a “teachers and teaching practices” component in 
PISA, an age/grade module would probably be more effective at age 9 than at age 15. At the primary level, 
many of the problems that make it difficult to disentangle the effects on older students’ achievement of 
teachers and teaching practices from other environmental factors are more easily manageable. Most 
students usually have only one teacher; the definition of what is a “class” and a “school” is much more 
straightforward than at the secondary level; and the content of the instruction received during the few 
previous years of school attendance is both easier to identify and somewhat more uniform from country to 
country. Additional optional instruments (such as the current parent questionnaire, or possibly a teacher 
questionnaire) would, again, make much more sense at age 9 than at age 15. Einally, an age/grade option at 
age 9 would probably help in facilitating some form of co-operation between PISA and the lEA PIRLS 
assessment, in order to prevent duplication of studies. 

38. One of the challenges of adding a younger cohort would be to ensure that the anchoring of the 
two sets of instruments is well conceived (both on theoretical and empirical grounds). This challenge 
would be the greater the larger the gap between the ages of the two assessments. This would require a 
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serious revision of the various PISA assessment frameworks, so that the targeting of the tests can he 
extended to younger populations, covering their still emergent capacities. Note that such a revision would 
have other positive outcomes as well. Some countries expressed concern that the PISA tests have 
insufficient “hottom” to measure accurately the proficiency of 15 years old students at very low levels of 
the scale, and that they do not cover enough “rudimentary” skills. 

39. If instruments can he developed with appropriate anchoring between the two populations, it 
would become possible to compare the levels of proficiency of older and younger students and to develop 
an indicator of average yearly progress in the subject areas for which assessments are covered in both 
target populations. Such an indicator would only be a gross estimate, but it would be interesting per se, and 
also useful as a tool for the interpretation of the magnitude of the effects of other factors. For example, 
when indicating that a difference of one unit of HISEI (or of any other independent variable) is associated 
with a difference of, say, 42 score points in reading, it would greatly help the reader to know that 42 score 
points corresponds to the average progress that is estimated to occur across the PISA countries in, let’s say, 
one school year (or 6 months of schooling, or some other duration of school time). 

40. Due to the fact that the PISA samples are based on age rather than on grades, the difference in 
achievement levels between the two populations would be easier to interpret in terms of relative efficiency 
of the various systems than when comparing two grade populations: it would be a measure of the gain 
obtained, on average, by comparable samples of students who spent six full school years in each system. 

41. Of particular interest would be a comparison of the variance decomposition between the two sets 
of data. At the primary level, the value of the between school variance component is usually lower than at 
the secondary level, because in the majority of countries all primary school teach equivalent curricula, and 
between-schools tracking only occurs at the secondary level. Thus the variance in achievement between 
primary schools is a relatively good indicator of the magnitude of mainly geographical and social 
aggregation phenomena (i.e. the fact that rural and urban schools, or private and public schools, etc, do not 
enrol the same proportions of children from privileged families). 

42. In many countries, the intraclass correlation becomes dramatically higher at the secondary level, 
indicating that other aggregation factors, such as differences between tracks and study programmes are at 
play in determining which students will attend which schools and the kind of instruction they will receive. 
This increase in between-schools variance is usually very large in countries with early curriculum 
differentiation. It can also be notable in some countries with officially comprehensive systems, but where 
“covert” selection is actually practiced across schools. Conversely, it is interesting to note that the 
between-schools variance tends to decrease somewhat in countries where all students share the same 
curriculum during the whole of ISCED 2 (particularly in some Nordic countries). It would be important to 
explore this issue further if a younger population is included in PISA, and to try to understand whether the 
latter phenomenon can be interpreted, in part, as a compensatory effect. 

43. Another relevant comparison made possible by the two populations would concern the rich 
battery of attitude scales used in PISA, as well as student learning strategies. 

44. As already mentioned, using an age/grade sampling design for the younger population would be 
rewarding (particularly in terms of analytic potential, but also in some cases in terms of cost-effectiveness 
and in terms of operational feasibility). In particular, in the (perhaps) majority of countries an age/grade 
design would involve only minor increases in the number of students to be sampled over and age-based 
design. Including a teacher questionnaire (be it linked or not to the individual students) would be much less 
problematic than at the secondary level, with relatively higher probability that the information collected 
has reliable relationships with achievement. 
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45. The proposed grade-based sampling design would have major advantages over the class-based 
designs used typically in lEA studies, since it would allow to adequately distinguish between the 
performance variation between students, classes within schools and schools. 

Optional Module Three: Linking student performance with the instructional context 

46. Countries have expressed interest in international comparisons of the effectiveness and efficiency 
of education systems. Such comparisons would be considerably enhanced by combining the current age- 
based design of PISA with a grade -based component. 

47. 64% of OECD respondents consider the integration of a grade -based assessment component into 
the PISA design relevant or very relevant (Question 4). This corresponds to experience with PISA 2006, 
where almost half of the OECD countries are already implementing a grade-based sampling component to 
better relate PISA results to their national institutional structure, to examine between-school performance 
variation in greater detail, or to provide a better framework for interpreting school-level and student-level 
contextual data. 

48. 82% of the countries also consider giving more emphasis to improving the collection of student- 
level contextual information (Question 5.1) and 59% giving more emphasis to developing contextual 
information on teachers and teaching by linking PISA and the OECD teacher survey as relevant or very 
relevant (Question 5.2). In contrast, providing emphasis to developing contextual information from 
parents, by integrating a parental questionnaire into the PISA design was found relevant or very relevant 
only by 44% of the respondents (Question 5.3). Relating PISA to a future strategy for assessing adult 
competencies was rated as relevant or very relevant by 52% of the respondents (Question 5.4). 

49. The optional Module Three A would seek to develop comparisons of the instructional context of 
learning outcomes through the establishment of a supplementary grade/class-based assessment. 

50. The optional Module Three B, that would be open for countries participating in Module 3A, 
would involve an extension of the student and school context questionnaires, as well as the collection of 
system-level data, where possible through INES. 

5 1 . Those countries implementing module three A could also implement an optional Module Three 
C that would consist of extensions to the sampling procedures and context questionnaires to facilitate the 
link between PISA and the OECD survey on teachers, teaching and learning, as well as the exploration of 
other data collection methods and in-depth studies seeking to relate student performance and teacher data. 

52. While the main emphasis of the 2009 OECD survey on teachers, teaching and learning will be on 
questions relating to the working environment for teachers, building on the policy issues identified in the 
OECD’s thematic review attracting, developing and retaining effective teachers, the 2012 teacher survey 
will seek to extend the picture by capturing differences in national approaches to teaching and learning, 
which would lend themselves to the type of analyses proposed here. This would also provide sufficient 
time to explore how a link between PISA and a teacher survey could be pursued in methodologically 
appropriate and reliable ways. 

53. In addition to analysing issues of educational effectiveness and efficiency through relating 
observed outcomes with relevant input and process variables at student, teacher and school levels. Module 
3 could extend the picture with the collection of system-level background data on some key policy levers 
generally perceived to be conducive to raising effectiveness and efficiency. Einally, the module could 
further explore the distribution of decision-making responsibilities across the various stakeholders in the 
different areas of decision-making. This is an area where PISA 2000 began with the development of a 
relevant questionnaire module on which successive surveys have progressively built. 
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The utility of the proposed design to meet PISA’s analytic objectives 

54. This section outlines how the core and optional modules of the future data strategy would speak 
to the key analytic objectives of PISA, namely to facilitate comparisons of: 

• The quality of learning outcomes; 

• Equality in learning outcomes and equity in learning opportunities; 

• The effectiveness and efficiency of educational processes; and 

• The impact of learning outcomes on social and economic well-heing. 

Comparing the quality of learning outcomes 

55. With the core module, PISA will continue to place the emphasis on cross-national comparisons 
of the cumulative yield of education towards the end of compulsory education, aiming to assess to what 
extent education systems succeed in ensuring that young adults acquire some of the key competencies and 
dispositions believed to contribute to the foundations for further learning and a successful transition into 
adult life. It is suggested maintaining this focus. 

56. The core module will also devote efforts to examining the distribution of learning outcomes and 
to gain a better understanding of the individual, institutional and systemic attributes that are associated 
with performance differences between individuals, institutions and countries. This is an area where 
strengthening the interpretative framework of the PISA proficiency levels in future cycles could further 
enhance policy relevance and where Modules 3a and 3b will provide important new insights. 

57. Einally, PISA has shown that there are major differences among countries in the extent to which 
student performance varies among schools, with performance variation between schools in some of the 
best performing countries amounting to less than 10 per cent of students overall performance variation - so 
that parents in these countries can rely on high and consistent performance standards across the entire 
education system - whereas in other countries two thirds of the OECD average performance variation 
originates at school and/or programme levels, often combined with only moderate overall performance. 
This is an aspect that is likely to gain in policy relevance, particularly as poor school performance is a 
growing policy concern in many countries, and the optional Module 3b will allow countries to better 
measure between-school variation in student performance and its relationship to the overall variation in 
student performance more reliably. 

58. The 2009 assessment will provide the first full trend analysis in reading literacy over the period 
2000-2009. The core module will allow examination of how improvements in the quality of educational 
outcomes in their country in the PISA assessment areas compare to improvements in other countries. This 
assessment of trends will not only be based on the two major assessments of reading in 2000 and 2009, but 
also draw on the two minor assessments of reading in 2003 and 2006. The latter is quite important since 
only four data points are likely to generate sufficiently reliable data on trends. Similar trend analyses would 
become available in 2012 for mathematics and in 2015 for science. 

59. The period of nine years is also judged appropriate to examine the impact of policies introduced 
in the wake of PISA 2000, and it is therefore suggested that this time period will strengthen the analytic 
power of PISA through better relating PISA with the INES collection of system-level information, 
including information on the institutional context and the policy strategies countries pursue to raise 
performance levels. The longer-term strategy should therefore strengthen the links between PISA and 
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INES. Provisions could be made, through triangulation, to correlate system-level information on intended 
policies with perceptions about their implementation at student and school levels. This could be pursued as 
part of the core-module as well. 

60. An assessment of where countries stand with equipping their children and citizens with key 
competencies will always remain an important objective for international assessments. However, beyond 
an examination of trends in learning outcomes over time, the policy relevance of international assessments 
will be enhanced by shifting some of the emphasis from benchmarking student performance across 
countries towards comparing learning progress within education systems. Module 2 is devoted to this 
aspect of the “quality” dimension of PISA. 

61. Einally, PISA 2006 is currently piloting a parental questionnaire in 16 countries. Such a parental 
questionnaire could generate an external opinion of the perceived quality of education and could bring with 
it significant gains in analytic power with modest cost and resource implications. However, it is not yet 
clear whether there is sufficient added value from pursuing this on an internationally comparative basis. 
Parents of the assessed students could, for example, shed light on: their expectations and aspirations of 
their children’s educational and attainment labour market outcomes, their overall strategy for their 
children’s education, their current and planned investment in their children’s educational future, parental 
influence at critical junctures of a student’s education (e.g. subject choice), parental attitudes towards 
subject areas and their perceived efficacy for deriving labour market outcomes, and parental involvement 
with the school. In fact, one of the thematic reports from PISA 2009 could examine how students’, 
schools’ and parents’ views on the quality of educational outcomes relate and how they match with the 
observed performance levels of students and schools. 

Comparing equality in learning outcomes and equity in learning opportunities 

62. PISA 2000 and PISA 2003 already devoted significant attention to questions of equity, in terms 
of how the socio-economic background of students and schools influences learning outcomes and what 
policy levers are associated with these outcomes. 

63. Given the high and increasing policy relevance of issues of equity, it would be conceivable to 
make questions of equality in learning outcomes and equity in learning opportunities one of the 
overarching themes for the core module of PISA 2009. Given increasing socio-economic diversity in 
OECD countries, and growing penalties for individuals with low competencies in labour markets and 
societies, it is likely that this will remain one of the key priorities for education policy development in the 
years to come and there are a number of reasons why the 2009 PISA assessment would be well placed to 
examine this area further: 

• The reading competencies that will be a major area of the 2009 assessment will also represent an 
analysis of equity-related issues, because deficiencies in language competencies are widely 
regarded as the heart of equity-related problems in schools. 

• In 2006, the OECD will have completed its thematic review of equity-related policies, including 
policies related to international migration, which could provide a useful policy framework for the 
development of the PISA 2009 assessment. 

• While a comparison between PISA 2000 and PISA 2003 shows that, in a few countries, 
significant improvements in learning outcomes have been achieved over a relatively short period 
of time, second-order relationships, such as the impact of socio-economic background on student 
performance typically change at a much slower pace. The 2009 assessment would probably be 
the first opportunity to meaningfully explore the extent to which equity-related aspects of 
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educational performance have changed and to relate this to corresponding changes in policy and 
practice. 

• The second OECD teacher survey (for details on benefits and costs, see documents 
EDU/EC/CERI(2005)1 and 2) which would collect its data in parallel to the 2009 PISA 
assessment, could provide useful insights on how schools and teachers respond to addressing 
heterogeneity in the student body with regard to student abilities, interests and socio-economic 
contexts. 

• The first OECD assessment of adult competencies, tentatively scheduled for 2010-11, would 
provide evidence on the economic and social consequences of low performance that would 
provide important background for a review of the policy implications of PISA results, 
particularly the proportions of students achieving the lowest proficiency levels. It would also 
provide a useful external reference point for the PISA proficiency levels: currently, PISA makes 
judgements about what level of competencies is required for individuals to successfully 
participate in societies based on the features of reading, mathematics and science that are 
believed to be relevant in adult life. With results from the adult assessment, it will become 
possible to validate these judgements with evidence on the economic and social returns to 
performance at the different proficiency levels. This is closely related to the policy theme 
discussed next: the impact of learning outcomes on social and economic well-being. 

64. In addition to these outcomes of the core module. Module 2 would make it possible to examine 
how the distribution of learning outcomes evolves at different levels of education; to what extent education 
systems succeed in moderating the impact of the economic, social and cultural capital of students and their 
families on performance as students grow older; and how learning pathways and institutional structures in 
different countries relate to such outcomes. 

65. Addressing such issues successfully through PISA would require further improvements in the 
context questionnaires, most notably in the questions that capture the socio-economic background of 
students and schools. Such development would need to be prioritised in the 2009 and 2012 assessments. 

Effectiveness and efficiency of education systems 

66. One of the main reasons why policy makers are interested in international comparisons of student 
performance is the insight they give in the differences in the performance of schools and education systems 
which reflect the effectiveness and efficiency of educational processes. Nevertheless, while showing 
important differences in the performance between education systems, international assessments have had 
limited success in providing insight into the policy levers associated with these differences. Building on the 
synergies of the different OECD survey instruments it may be possible to achieve improvements within the 
existing operational and financial constraints of PISA. Nevertheless, for reasons of feasibility the proposals 
here still focus more on system level differences in conjunction with collecting data on practices, rather 
than exploring questions of effectiveness at micro levels. 

67. It would be conceivable to make the effectiveness and efficiency of educational processes the 
overarching theme for the PISA 2012 assessment. There are at least two reasons why the 2012 PISA 
assessment would be well placed to address this area. Eirst of all, an analysis of questions of effectiveness 
and efficiency would lend itself particularly well to the subject area of mathematics, as the most school- 
bound subject covered by PISA, which will be the focus of the 2012 assessment. Second, providing a 
useful basis for analytic work in this area would require significant instrument development. 2012 may be 
the earliest point at which feasibility would have shifted the boundary between the “must haves’” and 
“easy wins’” sufficiently to obtain a critical mass in policy insights at reasonable costs. 
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68. Modules 3A, B and C will be devoted to addressing effectiveness and efficiency questions, by 
making it possible to relate observed outcomes with relevant input and process variables at student, teacher 
and school levels. Beyond this, it would also be possible to extend the picture with the collection of 
system-level background data on some key policy levers generally perceived to be conducive to raising 
effectiveness and efficiency. PISA would allow for the linking of such data on intended policies to their 
implementation at school level, as perceived by parents, teachers and schools and, of course, with observed 
learning outcomes. 

69. Finally, future PISA surveys would allow to advance existing analyses with regard to what 
distribution of decision-making responsibilities across the various stakeholders in the different areas of 
decision-making appears to be most conducive to encouraging performance orientation and local 
responsiveness while, at the same time, ensuring that learning opportunities remain accessible on an 
equitable basis. This is an area where PISA 2000 began with the development of a relevant questionnaire 
module on which successive surveys have progressively built. 

70. An assessment of the effectiveness of individual institutions would not be pursued by PISA. 
Impact of learning outcomes on social and economic well-being 

71. Little comparative evidence is available on the impact of competencies acquired at school on 
social and economic well-being at individual and societal levels; or the extent to which learning 
opportunities are connected coherently across educational institutions or between educational institutions 
and other forms of learning. 

72. A close connection between school-based assessments and OECD’s planned assessment of adult 
competencies should allow for a review of the competencies, aspirations and engagement of youth who are 
leaving initial education and are assuming responsibility in work and life, together with their impact on 
social and economic well-being. The latter could include individual outcomes such as successful 
integration into the labour market, employment status and earnings, participation in further learning, civic 
participation and education throughout the life cycle; as well as aggregate outcomes such as fostering 
economic growth and social participation. 
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Management structure for PISA 

73. The Strategic Development Group considered the design of the current management structure, 
which is described in Annex A, broadly adequate but recommended the improvement of its 
implementation, most importantly by evaluating bids for future PISA assessments on how they would 
strengthen the capacity of the PGB to establish and monitor project priorities by improving information 
flows between the Consortium and the PGB, by ensuring that the expert groups represent the range of 
views among OECD countries on the issues involved, and by making the composition and work of the 
expert groups as well as the Technical Advisory Group more transparent. In particular, the Strategic 
Development Group recommended: 

• Extending the “Eorum” to the major domain of each assessment cycle. 

• Appointing the Sampling Referee by the PGB and have it report directly to the PGB. 

• Specify the expected analytic outcomes clearly in the terms of reference and require bidders to 
specify quality assurance procedures for each domain and the context questionnaires to respond 
to analytical demands and to avoid cultural bias. 

• Enriching the composition of the PISA Technical Advisory Group to include content and 
questionnaire expertise, and have the TAG regularly report to the PGB. 

• Maintaining the focus of National Project Managers on project implementation but allowing for 
more exchange between National Project Managers and the PGB. 

• Establishing a closer link between the PGB and the Technical Advisory Group. To this end, 
technical issues that have policy implications will be brought to the attention to the PISA 
Governing Board. To this end, a report from the Technical Advisory Group would be regularly 
presented to the PISA Governing Board, clarifying the implications of technical issues and 
explaining the outcomes of technical decisions. The capacity to facilitate communication between 
the policy and technical levels would be an important criterion for the choice of the chair of the 
Technical Advisory Group. 

74. Note that changes to the roles and operation of the PISA Governing Board, PISA National 
Project Managers, the Expert Groups, the Technical Advisory Group and the OECD Secretariat will 
require unanimity in the PISA Governing Board, as stipulated in the OECD Council decision C(1997)176. 
Other parts of the management structure, including the composition and role of expert groups, could be 
changed using the normal decision-making procedures of the Governing Board. 
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ANNEX A: CURRENT MANAGEMENT STRUCTURE OE PISA 



75. This section recapitulates the existing management arrangements, as established hy Member 
countries [doc. ref. DEELSA/ED/CERUCD(97)4 and 6 and EXD/PCM/EDU(2003)28]. 

Management - current arrangements and issues for review 

76. Tthe management model that it established for PISA the PGB, formerly known as the Board of 
Participating Countries (BPC), sought to capitalise on the strengths of OECD countries and existing 
infrastructures at national and international levels and to encourage co-operation and development with 
national educational and statistical agencies. The intention of the PGB was to combine: 

• Access to political coverage at the level of the OECD; 

• Ownership by participating OECD countries during project design, implementation, analysis and 
reporting; and 

• The necessary power of decision making on a day-to-day basis, through a contracted Consortium 
of professional institutions that is selected by the PGB. 

77. The project exists within the framework of the OECD legal and financial requirements of a 
decentralised project. 

78. The following describes the roles and responsibilities of the different bodies in more detail and 
identifies issues that the participating countries may wish to review. 

PISA Governing Board (PGB) 

79. Through the PGB, countries determine, in the context of OECD objectives, the policy priorities 
for PISA and oversee adherence to these priorities during implementation. This includes the setting of 
priorities and standards for data development, analysis and reporting as well as the determination of the 
scope of work that will then form the basis for an international contractor. 

80. The PGB also works with the OECD Secretariat to ensure compliance with the policy objectives 
and design parameters at milestones during the implementation of PISA. In addition to enabling 
participating countries to share substantively with one another the programme’s decision-making and 
policy direction, the PGB enables participating countries to be fully informed of all aspects of PISA’s 
implementation. 

8 1 . The PGB is composed of representatives of the countries participating in PISA. Members of the 
PGB, designated by the governments they represent, need to be knowledgeable about large-scale student 
assessments and their interface with policy and practice. To ensure PISA’s overall coherence, continuity in 
the PGB’s membership across survey cycles is considered highly desirable. The PGB elects its chair for a 
period of three years. The PGB also elects, on a rotational basis, three vice chairs for a period of three 
years. 
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82. Partner countries participating in PISA are represented in the PGB as observers. This status 
entitles them to all materials and documents related to PISA and to participate in discussions at meetings of 
the PGB and its other deliberations. However, partner countries are not permitted to vote, in cases where 
the PGB chooses to vote on an issue where unanimity cannot be achieved. 

83. The PGB has established an Executive Group, consisting of the chair, the three vice chairs and 
the Secretariat, that facilitates the management of the programme. The Executive Group has an advisory 
role to the PGB and its recommendations need to be validated by the PGB through formal consultations. 

84. In 2002, the PGB also established a Strategic Development Group (SDG) to advise the PGB on 
the establishment of a broader analytic agenda for PISA that would extend across the various future survey 
cycles and, in consultation with the INES Network A, on the strategic design and development of PISA. 
The Strategic Development Group takes an advisory role vis a vis the PGB which retains decision-making 
responsibility on these matters. Eurthermore, all aspects requiring input from national authorities are dealt 
with through the PGB rather than through the Strategic Development Group. Members serve on the 
Strategic Development Group for renewable terms of 18 months and the PISA Governing Board will 
regularly review the composition of the group in light of changing priorities in the work programme. The 
Strategic Development Group elects its own chair and will be supported by the OECD Secretariat which 
will also convene and organise its meetings. 

85. Einally, the PGB established an Editorial Group in 2002 to guide and monitor the preparation of 
the PISA thematic reports. The composition and working arrangements of the Editorial Group are similar 
to those of the Strategic Development Group. 

86. The decision-making processes for PISA were established by the INES Network A in 1996 and 
formalised by the OECD Council in 1997 [doc. ref. C( 1997) 176]. Accordingly, the PGB shall seek 
consensus of all members in its deliberations and decisions. Decisions which are brought to a vote by 
either the chair or any of its members need to be adopted by a majority of participating members. Each 
year the PGB will prepare and approve by at least a two-thirds majority decision of the members the draft 
programme of work and budget for the following year. The PGB will, by a two-thirds majority decision of 
the members of the Board, also submit the annual estimates for expenditure to the OECD Budget 
Committee for adoption. 

87. Any necessary regulations and rules for the application of the rules governing its operation, 
changes to the scale of country appropriations to the budget as well as changes to the project design and 
structure described in document DEELSA/ED/CERI/CD(1997)4 need to be adopted by the PGB with a 
unanimous vote. 

National Project Managers 

88. Participating countries appoint National Project Managers to carry out the PISA surveys in the 
national context. These will interact with and report to the international contractor on all issues related to 
the implementation of the assessment in their country. Although National Project Managers will implement 
a coherent and well-defined project specified by the PISA Governing Board and the international 
contractor, they play a vital role in ensuring a high quality implementation and verifying and evaluating the 
survey results, analyses, reports and publications. 

89. National Project Managers are the primary means of day-to-day contact between participating 
countries and the contractor and shall communicate with this contractor on all issues related to the 
implementation of PISA in their country. 
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90. At the national level, each country decides how it can best facilitate the communication and 
co-ordination needed for implementing data collection responsibilities as well as for interacting with 
international contractor(s). 

National Committees 

91. Countries participating in PISA are expected to establish a national committee, drawn from the 
national educational and scientific community and experts involved in student assessment. These 
committees do, for example, provide advice to the national representatives to the PGB and to the National 
Project Manager on the appropriateness of the international data-collection instruments in the national 
context, on matters of quality control, and the dissemination of the results at the national level. 

Expert Groups 

92. Substantive input from Member countries is imperative in ensuring that the PISA assessments are 
internationally valid and take into account the cultural and curricular context of the countries in which they 
are implemented. The PISA management structure foresees that the international contractor, that will have 
ultimate responsibility for the implementation of PISA, works to ensure that participating Member 
countries are actively engaged in the development of the assessment instruments by establishing expert 
groups drawn from participating countries. These expert groups are expected to link the policy objectives 
specified by the PISA Governing Board with substantive and technical expertise and, under the guidance 
of the international contractor, to establish consensus on subject matter and technical issues among 
countries within the overall objectives and framework of the PISA strategy. 

93. The international contractor works with participating countries on nominations for the expert 
groups, encouraging wide country representation. At the same time, the establishment of the expert groups 
requires countries to delegate authority among themselves so as to keep the size of expert groups 
manageable and minimise costs. The limited size of the expert groups does not mean that only a limited 
number of countries will participate in the development of the project; rather, all countries are expected to 
contribute to developmental activities and to review the results of all components of the project although 
not necessarily each country will participate in each developmental activity. The international contractor 
proposes nominations to the Secretariat, which appoints the experts in consultation with the PISA 
Governing Board. Meetings of expert groups will be called and sponsored by the international contractor. 
Provisions for this have been made in the proposed budget. The Secretariat will decide on its own 
participation in expert groups on a case by case basis in consultation with the international contractor. 

94. In addition to the expert groups, an Open Forum for the main assessment area of each PISA 
assessment provides further opportunities, in particular for countries that are not represented on the 
relevant expert group, to participate in the development of the PISA assessments. 

Technical Advisory Group 

95. The international contractor must ensure the technical quality of the Project. The Technical 
Advisory Group assists this objective. As opposed to the subject matter expert groups, it has a permanent 
role across survey cycles and includes, among other experts, those individuals who have a leading 
operational role in the project operations such as, for example, major subcontractors. Thus, the Technical 
Advisory Group constitutes a forum through which the main actors implementing the project interact both 
among themselves and with those whose additional technical expertise is sought. 

96. The international contractor proposes nominations for the Technical Advisory Group to the 
Secretariat, which appoints the experts in consultation with the PISA Governing Board. 
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97. The alternative of setting the Technical Advisory Group up as an entirely external review panel 
was not envisaged in the management structure for PISA. However, the PISA Governing Board draws on 
additional scientific expertise for review purposes as needed. 

98. The international contractor raises technical issues that have policy implications with the 
Secretariat which, in turn, raises these with the PISA Governing Board as necessary. 

99. Meetings of the Technical Advisory Group are called and sponsored hy the international 
contractor. Provisions for this have been made in the proposed budget. 

100. Technical issues that have policy implications will be brought to the attention to the PISA 
Governing Board. To this end, a report from the Technical Advisory Group would be regularly presented 
to the PISA Governing Board, clarifying the implications of technical issues and explaining the outcomes 
of technical decisions. The capacity to facilitate communication between the policy and technical levels 
would be an important criterion for the choice of the chair of the Technical Advisory Group. 

Sampling referee 

101. The international contractor must ensure the integrity of national samples. For this purpose the 
PISA Governing Board appoints a Sampling Referee for which the international contractor will provide 
nominations. The international contractor will decide on the quality of the samples and their implications 
on the use of country results in the international reports on advice from the Sampling Referee. 

OECD Secretariat 

102. The OECD Secretariat, as agent of the PGB, is responsible for PISA’s overall management. This 
entails preparing the terms of reference for each survey cycle under the guidance of the PGB; engaging 
contractors to implement specified activities; and, monitoring the contractor(s) for quality assurance 
purposes and the project on a day-to-day basis. The OECD Secretariat also is responsible for building 
consensus among participating OECD countries at the policy level - through the PGB - both during the 
preparation of the terms of references and at milestone points of the surveys. 

103. The OECD Secretariat serves as the Secretariat of the PGB and as the interface between the PGB 
and the contractors during ail stages of PISA. It is a further responsibility of the OECD Secretariat to 
provide the PGB with a progress report on no less than a biannual basis as well as with a report on 
financial and contractual management on an annual basis. 

104. The OECD Secretariat produces the analyses, based on the statistical components provided by the 
contractor(s), and is responsible for preparing the international report in collaboration with the OECD 
countries, through the PGB, and the contractors. It is also responsible for overseeing the thematic reports, 
which are published under the responsibility of the OECD. The OECD Secretariat edits and revises draft 
reports submitted by contractors to ensure their suitability with respect to the interests of OECD countries 
in data and analysis that are responsive to their needs for information on policy and practice. 

105. OECD countries agree on a set of general rules for the inclusion/exclusion of country results in 
international reports. The OECD Secretariat arbitrates disagreements between participating countries and 
the sampling referee under guidelines established by the PGB. 
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International contractor charged with the implementation 

106. The design and implementation of the surveys, within the framework established hy the PISA 

Governing Board, is the responsibility of contractors selected through an open international call for tender. 

The main responsibilities of the contractor are to: 

• Prepare, and after receipt of comments from the PGB and OECD Secretariat, revise and complete 
assessment instruments; 

• Ensure the quality of the translation of all items relevant to the assessment of the three 
substantive domains plus all items or data-collection instruments in the national language(s) used 
in each participating country; 

• Work with countries to define the target population and draw the samples; 

• Establish survey procedures and operations, including field trialling and data collection in each of 
the three substantive domains and the context questionnaires; 

• Perform data verification and establish quality-control mechanisms; 

• Co-ordinate scaling and preparation of data products to be distributed to participating countries 
and to other contractors; 

• Create and manage the appropriate Subject Matter Expert Groups, notably in science; 

• Co-ordinate with National Project Managers in participating countries and the OECD Secretariat; 

• Provide technical and analytical support to the OECD Secretariat during the preparation of the 
international report, including the preparation of all tables and technical documentation; 

• Provide reasonable technical and analytical support to the authors of thematic reports; 

• Prepare, and after receipt of comments from the PGB and OECD Secretariat, revise and complete 
a framework for the collection of background or explanatory information on participating 
students, schools and perhaps teachers; 

• Develop questionnaires based on this framework; 

• Eield trial, analyse and revise the field trial questionnaires; 

• Develop an analysis and reporting plan that encompasses data collected through PISA; 

• Develop a detailed conceptual strategy that discusses at least two thematic areas (following the 
PGB’s priorities), how they will complement each other, and how each will contribute to an 
improved understanding of the policies and practices that affect educational outcomes; 

• Describe the process for developing the thematic reports {e.g., the processes for identifying and 
selecting authors, revision, quality control, and consensus building); 

• Analyse the data required to complete and then prepare in draft at least two thematic reports that 
are suitable for formal review by the PGB and the OECD Secretariat; 
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• Co-operate with the OECD Secretariat during the development of the initial international report, 
including providing descriptions, tables, or any results that might he relevant; 

107. The international contractor is also responsible for securing the skills and expertise necessary to 
conduct the specified activities; some possible means for doing so include using its own staff, making 
arrangements with government entities and contracting out some of the activities. In carrying out the 
project, the international contractor will ensure that: 



• Eunctional expertise is built into the project as needed, for example, by utilising appropriate sub- 
contractors; 

• Participating Member countries are actively engaged in the development of the assessment 
instruments through the expert groups; and 

• Consensus of participating Member countries is reached at the operational and technical level 
through National Project Managers. 



25 




